#####################
REPLICATION MATERIALS
#####################

AUTHORS: Neunhoeffer, Marcel & Sternberg, Sebastian
ARTICLE: "How Cross-Validation Can Go Wrong and What to Do about it."
JOURNAL: Political Analysis
UPDATED: June 26, 2018
CONTACT: marcel.neunhoeffer@gess.uni-mannheim.de; sebastian.sternberg@gess.uni-mannheim.de


INSTRUCTIONS:

To replicate the analysis, you first have to download the "man_ses_replication.zip" folder from the Dataverse. 


1. Unzip „man_ses_replication.zip“ - This will be the main folder.

2. Now you can get started by opening „Replication.Rmd“. (Which also contains further information about the replication files.)

SOFTWARE:

To open and run the „Replication.Rmd“ file you will need a current version of RStudio (we used RStudio 1.1.383). All the R packages needed for replication will be automatically installed and loaded when running „Replication.Rmd“.

SOFTWARE ENVIRONMENT:

platform 			x86_64-apple-darwin15.6.0 
arch 				x86_64
os 					darwin15.6.0
system 				x86_64, darwin15.6.0
status 
major 				3
minor 				4.3	
year 				2017
month 				11
day				30
svn rev 			73796
language 			R
version.string			R version 3.4.3 (2017-11-30)
nickname 			Kite-Eating Tree

RStudio				1.1.383

R PACKAGES:

- randomForest
- caret
- ROCR
- pROC
- stepPlr
- doMC
- separationplot
- logistf
- extrafont

DATAVERSE STRUCTURE (in the main folder):

• R -> contains R script files needed to run Replication.Rmd
	- setup.R -> setup of the working environment
  	- experiments.R -> code to run the experiments
  	- reanalysis.R -> code for the re-analysis of Muchlinski et al. (2016)
  	- AUC_PR.R -> functions to calculate the PR AUC, from Cranmer and Desmarais (2016)
• data
	- SambnisImp.csv - The replication data set provided by Muchlinski et al. (2016)
• Replication.Rmd -> This file, replicates everything in the article

• man_ses_online-appendix.pdf -> This is the Online Appendix of the paper.


RUNNING TIMES:

Setup (typical Laptop): MacBook Air 2014, 1.4 GHz Intel Dual-Core i5, 8GB RAM

Replication.Rmd ~ 12 minutes


NOTE:
If you have any further questions concerning the replication materials, please send an email to Marcel Neunhoeffer and/or Sebastian Sternberg. 
